Video editing is often performed by skilled personnel using expensive video editing software. It often requires significant time and effort, including reviewing and editing the many frames of the video.
Advances in camera and rendering technology have made it feasible to augment live video footage such as sports broadcasts, with virtual content, such as advertisements, or virtual content enabling tactical analysis. To generate a composite video with both the real and the virtual footage, knowledge of the scene structure and camera position/orientation is necessary. Some systems approach this challenge by engineering the recording conditions (i.e., hardware camera trackers in the tripods) and handcrafting the scene models. Overall this is a tedious, expensive and time-consuming process—and has questionable results unless certain controlled conditions are met.
Unfortunately, the required level of instrumentation makes such augmentation technology unusable for video footage recorded under more general conditions. Many existing solutions still require a fair amount of manual intervention and do not enable easy handling of difficult situations in general scenes, such as occlusion.
These and other challenges remain unsolved and, in some instances, unrecognized. The ability to automatically embed content in an efficient and flexible manner opens the door for a variety surprising applications and results.